[Backend Tester] Upload test report artifacts in CI job #13444

GregoryComer · 2025-08-15T00:09:26Z

Upload the test reports as part of the nightly job. This report contains the test case summary and details, such as accuracy statistics, PTE size, and lowering time.

[ghstack-poisoned]

GregoryComer · 2025-08-15T00:09:28Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2025-08-15T00:09:30Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13444

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 45 Unrelated Failures

As of commit 4d5f1d0 with merge base 5b743f5 ():

NEW FAILURES - The following jobs have failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold
nightly / backend-test-linux (xnnpack_static_int8_per_channel, operators) / linux-job (gh)
test_sqrt_edge_cases_xnnpack_static_int8_per_channel
nightly / backend-test-linux (xnnpack, operators) / linux-job (gh)
test_sqrt_edge_cases_xnnpack
trunk / test-huggingface-optimum-coreml (distilbert|coreml_fp32_gpu|--quantize) / macos-job (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-models-arm-zephyr (add) / linux-job (gh)
RuntimeError: Command docker exec -t abcca9cf1fb9ae4e4b091a2f5ab1936f2f4ce2d333fd9ec9cb2e626085c7141c /exec failed with exit code 1
trunk / test-models-arm-zephyr (mv2) / linux-job (gh)
RuntimeError: Command docker exec -t 83b0ccdbcad892eaf16c8b6ce739906c71b652acf7d876e71183691cb7a93e0a /exec failed with exit code 1
trunk / test-models-arm-zephyr (softmax) / linux-job (gh)
RuntimeError: Command docker exec -t dfe47eb66b86ac17ad53d08d253f94c725fc9922c7dc9c73769c77b21a996d70 /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

nightly / backend-test-linux (xnnpack_static_int8_per_channel, models) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
nightly / backend-test-linux (xnnpack, models) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
trunk / test-huggingface-optimum-coreml (olmo|coreml_fp32_gpu|--quantize) / macos-job (gh) (similar failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / unittest-buck / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / unittest-editable / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-coreml-delegate / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-custom-ops-macos (cmake) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-huggingface-optimum-coreml (bert|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
trunk / test-huggingface-optimum-coreml (llama3|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-huggingface-optimum-coreml (qwen3|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-huggingface-optimum-coreml (smollm|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-llama-torchao-lowbit / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (dl3) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (edsr) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (emformer_join) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (ic3) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (ic4) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (mobilebert) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (mv2) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (mv3) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (resnet50) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (vit) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (w2l) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (efficient_sam, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (emformer_join, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (llama, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (llama3_2_vision_encoder, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (mobilebert, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (mv3, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (mv3, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (resnet50, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-mps / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-selective-build-macos (cmake) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-static-llama-ane / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / unittest-release / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: d911d53 ghstack-comment-id: 3190250291 Pull-Request: #13444

[ghstack-poisoned]

Update

4d42d47

[ghstack-poisoned]

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 15, 2025

GregoryComer added a commit that referenced this pull request Aug 15, 2025

[Backend Tester] Upload test report artifacts in CI job

977cbfe

ghstack-source-id: d911d53 ghstack-comment-id: 3190250291 Pull-Request: #13444

This was referenced Aug 15, 2025

[Backend Tester] Add nightly CI job for XNNPACK #13390

Open

[Backend Tester] Mark adaptive avgpool2d as an unsupported portable op #13398

Open

[Backend Tester] Run Vulkan tests in nightly CI #13445

Open

Update

ad1ddaa

[ghstack-poisoned]

GregoryComer mentioned this pull request Aug 15, 2025

[Backend Tester] Run Core ML tests in nightly CI #13446

Open

GregoryComer and others added 4 commits August 14, 2025 18:15

Update

02c2e14

[ghstack-poisoned]

Update

5038454

[ghstack-poisoned]

Update

f070aa4

[ghstack-poisoned]

Update

4d5f1d0

[ghstack-poisoned]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Backend Tester] Upload test report artifacts in CI job #13444

[Backend Tester] Upload test report artifacts in CI job #13444

GregoryComer commented Aug 15, 2025 •

edited

Loading

Uh oh!

GregoryComer commented Aug 15, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 15, 2025 •

edited

Loading

Uh oh!

Uh oh!

[Backend Tester] Upload test report artifacts in CI job #13444

Are you sure you want to change the base?

[Backend Tester] Upload test report artifacts in CI job #13444

Conversation

GregoryComer commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GregoryComer commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13444

❌ 7 New Failures, 45 Unrelated Failures

Uh oh!

Uh oh!

GregoryComer commented Aug 15, 2025 •

edited

Loading

GregoryComer commented Aug 15, 2025 •

edited

Loading

pytorch-bot bot commented Aug 15, 2025 •

edited

Loading